Switch to new `flusurv` API endpoint #1278

nmdefries · 2023-08-23T17:21:59Z

Summary:

Pull flusurv data from new CDC API endpoint. Ingest previously unlabelled age groupings. Ingest new race/ethnicity and sex breakdowns. Test flusurv.py functions. Add more comments, messaging, and assertions.

Closes #1247
Closes #242

Note:

Some of our CDC contacts are able to provide us with past versions of data from the API. We will want to patch those in. Many of the "new" strata are actually available back to 2009. We'll probably want to patch those in as well.
This also requires updates to the database table, the API server, and the API docs that will be made separately.

Prerequisites:

Unless it is a documentation hotfix it should be merged against the dev branch
Branch is up-to-date with the branch to be merged with, i.e. dev
Build is successful
Code is cleaned up and formatted

- rename input arg to `update` to avoid reassignment later - comment and reuse args_insert - spelling - comment magic constant used in output format - rename location-network/catchmentid map

Previously, age strata were numbered sequentially which allowed us to store rate values by position in a list. With the introduction of the new strata, this system is not robust enough to track all the different groups (e.g. ageids are no longer sequential and there are now race and sex groupings with separate numbering systems).

…piweek

src/acquisition/flusurv/new_grasp_location_result.json

melange396

nice work! i really appreciate the thorough commenting!

youll wanna pull in the changes from the dev branch, PR #1241 added tests for the flusurv endpoint.

src/acquisition/flusurv/flusurv.py

src/acquisition/flusurv/flusurv_update.py

Co-authored-by: melange396 <george.haff@gmail.com>

aysim319 · 2025-01-16T15:41:20Z

ran the code locally and fails with

found data for 82 epiweeks
[successfully fetched data for network_all]
rows before: 0
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/src/app/delphi/epidata/acquisition/flusurv/flusurv_update.py", line 213, in <module>
    main()
  File "/usr/src/app/delphi/epidata/acquisition/flusurv/flusurv_update.py", line 204, in main
    update(fetcher, location, args.test)
  File "/usr/src/app/delphi/epidata/acquisition/flusurv/flusurv_update.py", line 155, in update
    raise Exception(
Exception: network_all 202401 data includes new group(s) {'rate_age_1t4', 'rate_age_gte75', 'rate_age_0tlt1'}

nmdefries · 2025-02-07T22:56:03Z

Exception: network_all 202401 data includes new group(s) {'rate_age_1t4', 'rate_age_gte75', 'rate_age_0tlt1'}

This should be resolved just by adding those signals to the list of those expected.

nmdefries · 2025-02-10T21:37:02Z

This works now. I added a migration .sql script to add the new columns to the schema.

nmdefries · 2025-02-10T21:37:43Z

@melange396 @aysim319 This is ready for review.

nmdefries · 2025-02-26T23:09:08Z

I duplicated some of the work in #1287 in c88c6fc. (#1287 needs to be re-merged into this PR anyway, see comment.)

nmdefries · 2025-02-26T23:10:26Z

These will need to be updated with new fields:

sonarqubecloud · 2025-02-27T00:04:07Z

Quality Gate passed

Issues
6 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
9.6% Duplication on New Code

See analysis details on SonarQube Cloud

nmdefries · 2025-10-08T19:43:15Z

Things to check before putting into prod:

DB migration (will need to test in staging)
interacting correctly with DB
naming new columns correctly and inserting them in the right spots
- avoid mixed column meanings, e.g. don't change meanings of col names
clients work

nmdefries · 2025-10-08T19:46:04Z

src/acquisition/flusurv/reference/new_grasp_result.json

@@ -0,0 +1,167 @@
+### New API response from https://gis.cdc.gov/GRASP/Flu3/PostPhase03DataTool?appVersion=Public


note: the red highlighting here seems to be because the JSON format is invalid (JSON doesn't allow comments and also maybe single quotes are disallowed). However, this file is for reference only, so it doesn't really matter.

brookslogan · 2025-10-09T16:15:29Z

Things to check before putting into prod:

naming new columns correctly and inserting them in the right spots

avoid mixed column meanings, e.g. don't change meanings of col names

I did a quick skim of the DB migration script and some of the acquisition code dealing with column names and ids. The former passed a sanity check and didn't seem like a risky operation anyway. The latter seems to have been written with a good deal of care (e.g., the changes to the overall age group column id and the new age group column ids). I don't have an environment to mock stuff but if you need a set of eyes to sanity-check the acquisition result after deployment, feel free to ping me and I can double-check API vs. upstream values.

nmdefries added 11 commits September 14, 2023 17:47

initial switch to new API endpoint; doesn't account for format change

d99d2d0

separate fetch fn for whole api obj

70c3e16

get_current_issue to use existing json response

3580f3f

cleanup names and comments

9cb1da4

- rename input arg to `update` to avoid reassignment later - comment and reuse args_insert - spelling - comment magic constant used in output format - rename location-network/catchmentid map

define function to convert json obs to dict grouped by location and e…

cea25ab

…piweek

auto-map from valueids to ordinal and label-based group names

24dc088

add new strata to sql insert statement by name, not order

26eff97

pass seasonids around to use in requests for location-specific data

229a96c

include old and new example API responses

1e942a5

flusurv tests

f8a6706

nmdefries force-pushed the ndefries/flusurv-new-endpoint branch from 734dca9 to f8a6706 Compare September 15, 2023 15:50

Merge branch 'dev' into ndefries/flusurv-new-endpoint

d9581ac

nmdefries marked this pull request as ready for review September 15, 2023 16:25

nmdefries commented Sep 15, 2023

View reviewed changes

src/acquisition/flusurv/new_grasp_location_result.json Show resolved Hide resolved

nmdefries added 2 commits September 15, 2023 16:07

pass metadata around to reduce API calls

9ac2918

add season label as a descriptive column

aa3dd8c

nmdefries requested a review from brookslogan September 15, 2023 20:50

move example API responses to make it clear they are not for prod use

ca7e478

nmdefries removed the request for review from brookslogan September 15, 2023 21:51

nmdefries mentioned this pull request Sep 15, 2023

Update flusurv schema and docs with new age, sex, and race groups #1287

Merged

4 tasks

nmdefries requested a review from melange396 September 18, 2023 15:46

This was referenced Sep 18, 2023

flusurv request is missing columns cmu-delphi/epidatpy#25

Open

flusurv request is missing columns cmu-delphi/epidatr#180

Open

melange396 requested changes Sep 26, 2023

View reviewed changes

nmdefries and others added 5 commits September 27, 2023 13:22

Merge branch 'dev' into ndefries/flusurv-new-endpoint

b979910

review cleanup

c1614a1

capitalize constant max_age

19a5b42

move paren to subtract from # of dates

2f18451

Co-authored-by: melange396 <george.haff@gmail.com>

move n_exected_groups and big groupid comment to global

d64c4c0

nmdefries mentioned this pull request Dec 19, 2024

Better describe meaning/context of endpoints in documentation cmu-delphi/epidatr#233

Open

7 tasks

aysim319 self-requested a review January 16, 2025 15:41

add new age rate signals

cf2fcec

nmdefries added 3 commits February 10, 2025 13:54

add new columns to flusurv endpoint spec

16634f8

add schema migration script

8be1074

Merge branch 'dev' into ndefries/flusurv-new-endpoint

87ecfdb

nmdefries added 12 commits February 12, 2025 18:54

move base url to constants

76c0f00

add max age as CLI arg

c9f279f

add flu a/b to DB table

9dd1435

hardcode valueid->signal name; add flu signals

a93e3b1

remove ordinal conditional blocks from _groupid_to_name definition

059d7c5

add back female col to sql schema

9be73a5

make error types more specific

9db5935

add new cols to integration test

e651646

remove trailing comma

6c80297

specify flusurv return fields

c88c6fc

warn on unexpected or missing keys and fill in

ab41fa0

use unexpected key's ids as suffix; drop UT sex group handling

5938b46

nmdefries added 2 commits February 26, 2025 18:19

cast ids to str

38c6338

drop trailing location

de9a277

nmdefries commented Oct 8, 2025

View reviewed changes

		@@ -0,0 +1,167 @@
		### New API response from https://gis.cdc.gov/GRASP/Flu3/PostPhase03DataTool?appVersion=Public

Switch to new flusurv API endpoint #1278

Are you sure you want to change the base?

Switch to new flusurv API endpoint #1278

Uh oh!

Conversation

nmdefries commented Aug 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary:

Prerequisites:

Uh oh!

Uh oh!

melange396 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aysim319 commented Jan 16, 2025

Uh oh!

nmdefries commented Feb 7, 2025

Uh oh!

nmdefries commented Feb 10, 2025

Uh oh!

nmdefries commented Feb 10, 2025

Uh oh!

nmdefries commented Feb 26, 2025

Uh oh!

nmdefries commented Feb 26, 2025

Uh oh!

sonarqubecloud bot commented Feb 27, 2025

Quality Gate passed

Uh oh!

nmdefries commented Oct 8, 2025

Uh oh!

nmdefries Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brookslogan commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Switch to new `flusurv` API endpoint #1278

Switch to new `flusurv` API endpoint #1278

nmdefries commented Aug 23, 2023 •

edited

Loading

nmdefries Oct 8, 2025 •

edited

Loading

brookslogan commented Oct 9, 2025 •

edited

Loading